Interactive Display of Aggregated Search Result Information

ABSTRACT

A method, system, and processor-readable storage medium are directed towards generating a report derived from data, such as event data, stored on a plurality of distributed nodes. In one embodiment the analysis is generated using a “divide and conquer” algorithm, such that each distributed node analyzes locally stored event data while an aggregating node combines these analysis results to generate the report. In one embodiment, each distributed node also transmits a list of event data references associated with the analysis result to the aggregating node. The aggregating node may then generate a global ordered list of data references based on the list of event data references received from each distributed node. Subsequently, in response to a user selection of a range of global event data, the report may dynamically retrieve event data from one or more distributed nodes for display according to the global order.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation patent application of ApplicationSer. No. 14/158,421, filed Jan. 17, 2014, which claims benefit as aContinuation of Ser. No. 13/660,845, filed Oct. 25, 2012, which claimsbenefit as a Continuation of Ser. No. 13/223,167, filed on Aug. 31,2011, which claims benefit of Provisional Patent Application Ser. No.61/452,591 filed on Mar. 14, 2011, the entire contents of theaforementioned are hereby incorporated by reference as if fully setforth herein, under 35 U.S.C. §120. The applicant(s) hereby rescind anydisclaimer of claim scope in the parent application(s) or theprosecution history thereof and advise the USPTO that the claims in thisapplication may be broader than any claim in the parent application(s).

TECHNICAL FIELD

The present invention relates generally to generating reports and moreparticularly but not exclusively to reducing latency when generatinginteractive reports from data contained on a plurality of distributedcomputing nodes.

BACKGROUND

An increasing number of computing applications, particularly within theenterprise, entail analyzing distributed data. One type of analysis isreport generation, such as generating a table, a chart, or a timelinefrom distributed data. Some reports are generated by copying all of thedata from the plurality of distributed nodes to a single aggregatingnode for analysis. However, if the amount of data to be analyzed islarge, then transmitting this data over a network can be prohibitivelytime consuming. Also, a single aggregating node may not contain thecapacity necessary to store data received from a plurality ofdistributed nodes. Accordingly, efficiently generating reports thatcontain aggregate information as well as raw data is an ongoingchallenge.

BRIEF DESCRIPTION OF THE DRAWINGS

Non-limiting and non-exhaustive embodiments are described with referenceto the following drawings. In the drawings, like reference numeralsrefer to like parts throughout the various figures unless otherwisespecified.

For a better understanding of the described embodiments, reference willbe made to the following Detailed Description, which is to be read inassociation with the accompanying drawings, wherein:

FIG. 1 illustrates a system diagram of an environment in whichembodiments of the invention may be implemented;

FIG. 2 illustrates an embodiment of a client device that may be includedin a system such as that shown in FIG. 1;

FIG. 3 illustrates an embodiment of a network device that may beincluded in a system such as that shown in FIG. 1;

FIG. 4 illustrates a logical flow diagram showing one embodiment of aprocess for generating and displaying an interactive report;

FIG. 5 illustrates a logical flow diagram showing one embodiment of aprocess for analyzing raw data on a distributed node for display in aninteractive report; and

FIG. 6 illustrates one non-limiting embodiment of an interactive report.

DETAILED DESCRIPTION OF THE INVENTION

Throughout the specification and claims, the following terms take themeanings explicitly associated herein, unless the context clearlydictates otherwise. The phrase “in one embodiment” as used herein doesnot necessarily refer to the same embodiment, though it may.Furthermore, the phrase “in another embodiment” as used herein does notnecessarily refer to a different embodiment, although it may. Thus, asdescribed below, various embodiments of the invention may be readilycombined, without departing from the scope or spirit of the invention.

In addition, as used herein, the term “or” is an inclusive “or”operator, and is equivalent to the term “and/or,” unless the contextclearly dictates otherwise. The term “based on” is not exclusive andallows for being based on additional factors not described, unless thecontext clearly dictates otherwise. In addition, throughout thespecification, the meaning of “a,” “an,” and “the” include pluralreferences. The meaning of “in” includes “in” and “on.”

As used herein, the term “event data” refers to computing data that iscollected about a computing system, including, for example, an action,characteristic, condition (or state), or state change of the computingsystem. For example, such events may be about a computing system'sperformance, actions taken by the computing system, or the like. Eventdata may be obtained from various computing log files generated by thecomputer's operating system, and/or other monitoring application.However, event data is not restricted by a file format or structure fromwhich the event data is obtained.

As used herein, an event record refers to data associated with a singleevent.

As used herein, the term “report” refers to one or more visualizationsof search query results. For example, a report may include a table ofdata, a timeline, a chart, a “field picker” or the like. In oneembodiment, the report is interactive, enabling a user to selectivelyview pieces of raw data used to generate the report. For example, if thereport lists users sorted based on the number of times each user haslogged into the system, each user is selectable to view detailed recordsof that user's login events.

Briefly described is a mechanism for generating a report derived fromdata, such as event data, stored on a plurality of distributed nodes. Inone embodiment the analysis is generated using a “divide and conquer”algorithm, such that each distributed node analyzes locally stored eventdata while an aggregating node combines these analysis results togenerate the report. In one embodiment, each distributed node alsotransmits a list of event data references associated with the analysisresult to the aggregating node. The aggregating node may then generate aglobal ordered list of data references based on the list of event datareferences received from each distributed node. Subsequently, inresponse to a user selection of a range of global event data, the reportmay dynamically retrieve event data from one or more distributed nodesfor display according to the global order.

Illustrative Operating Environment

FIG. 1 shows components of one embodiment of an environment in which theinvention may be practiced. Not all the components may be required topractice the invention, and variations in the arrangement and type ofthe components may be made without departing from the spirit or scope ofthe invention. As shown, system 100 of FIG. 1 includes local areanetworks (“LANs”)/wide area networks (“WANs”)-(network) 107, clientdevices 101-103, and distributed search server 109.

One embodiment of client devices 101-103 is described in more detailbelow in conjunction with FIG. 2. Generally, however, client devices101-103 may include virtually any computing device capable ofcommunicating over a network to send and receive information, includinga search query, analysis results of a search query, lists of event datareferences, collections of event data, and the like. Client devices101-103 are referred to interchangeably herein as “distributed computingdevices”, “distributed nodes”, or the like. In one embodiment, one ormore of client devices 101-103 may be configured to operate within abusiness or other entity to perform a variety of services for thebusiness or other entity. For example, client devices 101-103 may beconfigured to operate as a web server, an accounting server, aproduction server, an inventory server, or the like. However, clientdevices 101-103 are not constrained to these services and may also beemployed, for example, as an end-user computing node, in otherembodiments. Further, it should be recognized that more or less clientdevices may be included within a system such as described herein, andembodiments are therefore not constrained by the number or type ofclient devices employed.

The set of such client devices 101-103 may include devices thattypically connect using a wired or wireless communications medium suchas personal computers, servers, multiprocessor systems,microprocessor-based or programmable consumer electronics, network PCs,or the like. In one embodiment, at least some of client devices 101-103may operate over wired and/or wireless network. In some embodiments,client devices 101-103 may include virtually any portable computingdevice capable of receiving and sending a message over a network, suchas network 107.

Client devices 101-103 also may include at least one client applicationthat is configured to capture and record event data and/or relatedmetadata. However, the client application need not be limited to merelyproviding event data and related metadata, and may also provide otherinformation, and/or provide for a variety of other services, including,for example, monitoring for events within and/or between client devices.

The client application may further provide information that identifiesitself, including a type, capability, name, and the like. Suchinformation may be provided in a network packet, or the like, sentbetween other client devices, distributed search server 109, or othercomputing devices.

Network 107 is configured to couple network devices with other computingdevices, including distributed search server 109 and client devices101-103. Network 107 is enabled to employ any form of computer readablemedia for communicating information from one electronic device toanother. Also, network 107 can include the Internet in addition to localarea networks (LANs), wide area networks (WANs), direct connections,such as through a universal serial bus (USB) port, other forms ofcomputer-readable media, or any combination thereof. On aninterconnected set of LANs, including those based on differingarchitectures and protocols, a router acts as a link between LANs,enabling messages to be sent from one to another. In addition,communication links within LANs typically include twisted wire pair orcoaxial cable, while communication links between networks may utilizeanalog telephone lines, full or fractional dedicated digital linesincluding T1 , T2, T3, and T4, and/or other carrier mechanismsincluding, for example, E-carriers, Integrated Services Digital Networks(ISDNs), Digital Subscriber Lines (DSLs), wireless links includingsatellite links, or other communications links known to those skilled inthe art. Moreover, communication links may further employ any of avariety of digital signaling technologies, including without limit, forexample, Digital Signal (DS)-0, DS-1, DS-2, DS-3, DS-4, Optical Carrier(OC)-3, OC-12, OC-48, or the like. Furthermore, remote computers andother related electronic devices could be remotely connected to eitherLANs or WANs via a modem and temporary telephone link. In oneembodiment, network 107 may be configured to transport information of anInternet Protocol (IP). In essence, network 107 includes anycommunication method by which information may travel between computingdevices.

Additionally, communication media typically embodies computer-readableinstructions, data structures, program modules, or other transportmechanism and includes any information delivery media. By way ofexample, communication media includes wired media such as twisted pair,coaxial cable, fiber optics, wave guides, and other wired media andwireless media such as acoustic, Radio Frequency (RF), infrared, andother wireless media.

In some embodiments, network 107 may be further configurable as awireless network, which may further employ a plurality of accesstechnologies including 2nd (2G), 3rd (3G), 4th (4G) generation radioaccess for cellular systems, WLAN, Wireless Router (WR) mesh, and thelike. In one non-limiting example, network 107, when configured as awireless network, may enable a radio connection through a radio networkaccess such as Global System for Mobile communication (GSM), GeneralPacket Radio Services (GPRS), Enhanced Data GSM Environment (EDGE),Wideband Code Division Multiple Access (WCDMA), and the like.

Distributed search server 109 includes virtually any network deviceusable to receive a search query, distribute sub-queries of the searchquery among client devices 101-103, synthesize the results of thesub-queries, and display a report. Distributed search server 109 may,for example, be configured to merge lists of event data references intoa global ordered list of event data references, enabling ranges of eventdata to be selectively retrieved from one or more distributed nodes.

Devices that may operate as distributed search server 109 includevarious network devices, including, but not limited to personalcomputers, desktop computers, multiprocessor systems,microprocessor-based or programmable consumer electronics, network PCs,server devices, network appliances, and the like.

Although FIG. 1 illustrates distributed search server 109 as a singlecomputing device, the invention is not so limited. For example, one ormore functions of the distributed search server 109 may be distributedacross one or more distinct network devices. Moreover, distributedsearch server 109 is not limited to a particular configuration. Thus, inone embodiment, distributed search server 109 may contain a plurality ofnetwork devices to perform digest aggregation and calculation ofapproximate order statistics therefrom. Similarly, in anotherembodiment, distributed search server 109 may operate as a plurality ofnetwork devices within a cluster architecture, a peer-to-peerarchitecture, and/or even within a cloud architecture. Thus, theinvention is not to be construed as being limited to a singleenvironment, and other configurations, and architectures are alsoenvisaged.

Illustrative Client Device

FIG. 2 shows one embodiment of client device 200 that may be included ina system implementing embodiments of the invention. Client device 200may include many more or less components than those shown in FIG. 2.However, the components shown are sufficient to disclose an illustrativeembodiment for practicing the present invention. Client device 200 mayrepresent, for example, one embodiment of at least one of client devices101-103 of FIG. 1.

As shown in the figure, client device 200 includes processing unit (CPU)202 in communication with a mass memory 226 via a bus 234. Client device200 also includes a power supply 228, one or more network interfaces236, an audio interface 238, a display 240, and an input/outputinterface 248. Power supply 228 provides power to client device 200.

Network interface 236 includes circuitry for coupling client device 200to one or more networks, and is constructed for use with one or morecommunication protocols and technologies including, but not limited to,global system for mobile communication (GSM), code division multipleaccess (CDMA), time division multiple access (TDMA), user datagramprotocol (UDP), transmission control protocol/Internet protocol(TCP/IP), Short Message Service (SMS), general packet radio service(GPRS), Wireless Application Protocol (WAP), ultra wide band (UWB),Institute of Electrical and Electronics Engineers (IEEE) 802.16Worldwide Interoperability for Microwave Access (WiMax), SessionInitiation Protocol (SIP)/Real-time Transport Protocol (RTP), or any ofa variety of other communication protocols. Network interface 236 issometimes known as a transceiver, transceiving device, or networkinterface card (NIC).

Audio interface 238 is arranged to produce and receive audio signalssuch as the sound of a human voice. For example, audio interface 238 maybe coupled to a speaker and microphone (not shown) to enabletelecommunication with others and/or generate an audio acknowledgementfor some action. Display 240 may be a liquid crystal display (LCD), gasplasma, light emitting diode (LED), or any other type of display usedwith a computing device. Display 240 may also include a touch sensitivescreen arranged to receive input from an object such as a stylus or adigit from a human hand.

Client device 200 also comprises input/output interface 248 forcommunicating with external devices, such as a keyboard, or other inputor output devices not shown in FIG. 2. Input/output interface 248 canutilize one or more communication technologies, such as USB, infrared,Bluetooth™, or the like.

Mass memory 226 includes a Random Access Memory (RAM) 204, a Read OnlyMemory (ROM) 222, and other storage means. Mass memory 226 illustratesan example of computer readable storage media (devices) for storage ofinformation such as computer readable instructions, data structures,program modules or other data. Mass memory 226 stores a basicinput/output system (“BIOS”) 224 for controlling low-level operation ofclient device 200. The mass memory also stores an operating system 206for controlling the operation of client device 200. It will beappreciated that this component may include a general-purpose operatingsystem such as a version of UNIX, or LINUX, or a specialized clientcommunication operating system such as Windows Mobile™, or the Symbian®operating system. The operating system may include, or interface with aJava virtual machine module that enables control of hardware componentsand/or operating system operations via Java application programs.

Mass memory 226 further includes one or more data storage 208, which canbe utilized by client device 200 to store, among other things,applications 214 and/or other data. For example, data storage 208 mayalso be employed to store information that describes variouscapabilities of client device 200. The information may then be providedto another device based on any of a variety of events, including beingsent as part of a header during a communication, sent upon request, orthe like. At least a portion of the information may also be stored on adisk drive or other computer-readable storage device 230 within clientdevice 200. Data storage 208 may further store event data and metadata210 and local search results 212. Such event data and metadata 210 andlocal search results 212 may also be stored within any of a variety ofother computer-readable storage devices, including, but not limited to ahard drive, a portable storage device, or the like, such as illustratedby computer-readable storage device 230.

Applications 214 may include computer executable instructions which,when executed by client device 200, transmit, receive, and/or otherwiseprocess network data. Other examples of application programs includecalendars, search programs, email clients, IM applications, SMSapplications, Voice Over IP (VOIP) applications, contact managers, taskmanagers, transcoders, database programs, word processing programs,security applications, spreadsheet programs, games, search programs,data log recording programs, and so forth. Applications 214 may include,for example, local search module 220. Local search module 220 mayprocess a sub-query, returning analysis results and a list of event datareferences associated with the analysis results, as described herein.

Illustrative Network Device

FIG. 3 shows one embodiment of a network device 300, according to oneembodiment of the invention. Network device 300 may include many more orless components than those shown. The components shown, however, aresufficient to disclose an illustrative embodiment for practicing theinvention. Network device 300 may be configured to operate as a server,client, peer, or any other device. Network device 300 may represent, forexample distributed search server 109 of FIG. 1.

Network device 300 includes processing unit 302, an input/outputinterface 332, video display adapter 336, and a mass memory, all incommunication with each other via bus 326. The mass memory generallyincludes RAM 304, ROM 322 and one or more permanent mass storagedevices, such as hard disk drive 334, tape drive, optical drive, and/orfloppy disk drive. The mass memory stores operating system 306 forcontrolling the operation of network device 300. Any general-purposeoperating system may be employed. Basic input/output system (“BIOS”) 324is also provided for controlling the low-level operation of networkdevice 300. As illustrated in FIG. 3, network device 300 also cancommunicate with the Internet, or some other communications network, vianetwork interface unit 330, which is constructed for use with variouscommunication protocols including the Transmission ControlProtocol/Internet Protocol (TCP/IP) protocol. Network interface unit 330is sometimes known as a transceiver, transceiving device, or networkinterface card (NIC).

Network device 300 also comprises input/output interface 332 forcommunicating with external devices, such as a keyboard, or other inputor output devices not shown in FIG. 3. Input/output interface 332 canutilize one or more communication technologies, such as USB, infrared,Bluetooth™, or the like.

The mass memory as described above illustrates another type ofcomputer-readable media, namely computer-readable storage media and/orprocessor-readable storage medium. Computer-readable storage media(devices) may include volatile, nonvolatile, removable, andnon-removable media implemented in any method or technology for storageof information, such as computer readable instructions, data structures,program modules, or other data. Examples of computer readable storagemedia include RAM, ROM, EEPROM, flash memory or other memory technology,Compact Disc ROM (CD-ROM), digital versatile disks (DVD) or otheroptical storage, magnetic cassettes, magnetic tape, magnetic diskstorage or other magnetic storage devices, or any other non-transitoryphysical medium which can be used to store the desired information andwhich can be accessed by a computing device.

As shown, data storage 308 may include a database, text, spreadsheet,folder, file, or the like, that may be configured to maintain and storeuser account identifiers, user profiles, email addresses, IM addresses,and/or other network addresses; or the like. Data stores 308 may furtherinclude program code, data, algorithms, and the like, for use by aprocessor, such as central processing unit (CPU) 302 to execute andperform actions. In one embodiment, at least some of data store 308might also be stored on another component of network device 300,including, but not limited to computer-readable storage medium 328, harddisk drive 334, or the like. Data storage 308 may further store orderedlist of event data references 310.

Ordered list of event data references 310 may include a list of eventdata references received from a plurality of distributed nodes. In oneembodiment, the ordered list of event data references is generated bysorting data references received from each distributed node according toa common field, such as a timestamp, a number a string, or the like. Inone embodiment, each element of the ordered list includes a reference tothe distributed node the event data is stored on, an offset or otherpointer to the event data on that distributed node, and optionally thevalue used to sort the ordered list.

The mass memory also stores program code and data. One or moreapplications 314 are loaded into mass memory and run on operating system306. Examples of application programs may include transcoders,schedulers, calendars, database programs, word processing programs,Hyper Text Transfer Protocol (HTTP) programs, customizable userinterface programs, Internet Protocol Security (IPSec) applications,encryption programs, security programs, SMS message servers, accountmanagers, and so forth. Distributed search module 318 may also beincluded as application programs within applications 314.

Distributed search module 318 may be configured and arranged to receivea query, generate sub-queries for each of a specified set of distributeddevices, and aggregate results of these sub-queries to generate areport, as described further herein.

Generalized Operation

The operation of certain aspects will now be described with respect toFIGS. 4-6. FIGS. 4-5 provide logical flow diagrams illustrating certainaspects, while FIG. 6 illustrates an example of a scalable interactivedisplay of distributed data. FIG. 4 illustrates a logical flow diagramof one embodiment of a process for generating and displaying aninteractive report. In one embodiment, process 400 may be implemented ondistributed search server 109.

Process 400 begins, after a start block, at block 402, where a searchquery (hereinafter “query”) is received. In one embodiment, the receivedquery targets data, such as “event data” (also referred to as “events”),that is distributed across a plurality of specified computing devices,such as client devices 101-103. In one embodiment, sub-queries aregenerated for each of the specified computing devices and submitted toeach corresponding computing device for processing. For example, if thereceived query asks for a count of system log entries that contain theword “error”, then a sub-query is generated for each of the specifiedcomputing devices, where each sub-query counts the number of eventsderived from system log entries that contain the word “error” stored onthat device.

The received query may specify which computing devices to search in anumber of ways. In one embodiment, the received query specifiesparticular computing devices or groups of computing devices by name,network address, or the like. In another embodiment, computing devicesare specified based on attributes, such as operating system, hardwarecomponents (e.g. CPU, web cam, network adapter, etc.), form factor (e.g.laptop, desktop, server, tablet, virtual machine, smartphone, etc.), andthe like. In another embodiment, a query may specify all of theplurality of computing devices.

In one embodiment, the received query is received from a user, such as asystem administrator. However, queries may also be automaticallygenerated by a software agent. In one embodiment, a query may beautomatically generated at periodic intervals, such as every hour orevery Saturday night. In another embodiment, a query may be generated inresponse to an event, such as installation of a software patch, or inresponse to a metric crossing a threshold, such as an unusually largevolume of network traffic.

The distributed data to be searched may be stored on the specifiedcomputing devices in many ways. In one embodiment, the distributed datamay include events, as defined herein, that have been recorded andstored by each of the specified computing devices. However, thedistributed data may be generated at any time and in any manner,including partitioning a data set across specified computing devicesafter the query has been received. Also, while in one embodiment thedistributed data comprises “events” or “event data” as defined herein,the distributed data may include any kind of data, structured orunstructured.

The received query may include one or more analyses to be performed onthe distributed event data by the computing devices storing that data.For example, an analysis may include counting the number of events thatsatisfy a condition, deriving statistical information about events(including distributions, histograms, N^(th) percentile rankings, andthe like), grouping events, sorting events, and the like. That is, theanalysis may be performed in response to the query.

In one embodiment, the received query may also specify the order ofquery results. For example, a query requesting system log entries thatcontain the word “error” may be ordered based on the time the system logentry was generated (timestamp). A similar query may order entries basedon an error severity value field in the event derived from the systemlog entry. Multiple orderings and nested orderings are alsocontemplated, such as ordering first by an error severity value and thenby a timestamp.

The process proceeds to block 404, where sub-query results are receivedfrom each of the specified computing devices. In one embodiment, thesub-query results include analysis results corresponding to the one ormore analyses specified in the received query. In one embodiment,analysis results are derived from raw event data stored on each of thespecified devices, but analysis results do not include the actual rawevent data.

The sub-query results additionally include one or more lists of eventreferences. In one embodiment, each event reference includes (1) anidentifier that uniquely identifies the event on the computing devicethat generated it, and (2) a value usable to order the event(hereinafter “order value”). In one embodiment the unique identifierincludes a serial number assigned to the event as the event is created.In this example, the unique identifier is unique to a given computingdevice—events from different computing devices may be assigned the sameunique identifier. However, globally unique identifiers, such as GUIDs,are similarly contemplated.

In one embodiment, the order value of an event may be a timestamp, suchas the time when an event was created. However, any value of any datatype is similarly contemplated. Other examples of order values includeintegers, such as a severity of error, and a string, such as a username.

In one embodiment, a computing device creates an event reference foreach event used to generate an analysis result. For example, considerthree devices A, B, and C that contain 14, 37, and 94 system log eventscontaining the word, “error”, respectively. If a query to count all ofthe system log events that contain the word “error” is received, deviceA will return a count of 14 as well as a list of 14 references, onereference for each of the 14 counted events. Similarly, device B willreturn a count of 37 as well as a list of 37 references, and device Cwill return a count of 94 and a list of 94 references. Note that at thistime, none of the raw event data has been transmitted to the distributedsearch server.

The process proceeds to block 406, where a global ordered list of eventreferences is generated based on each of the returned lists of eventreferences. In one embodiment, each entry in the global ordered listincludes the content of an event reference, as described above, as wellas an identifier of the computing device that the event was found on.

Continuing the example above, consider if the first 7 of device A'sevents were the first to be recorded, followed by the first 50 of deviceC's, followed by all 37 of device B's, followed by the last 44 of deviceC, and finally the last 7 of device A. In this simple example, theglobal ordered list would include all 145 event references in this sameorder, where each event reference is fully qualified to include acomputing device identifier in addition to that event's uniqueidentifier. In this way, a user may select a range from the globalordered list of event references, and only the actual event datacontained in the selected range is downloaded.

The process proceeds to block 408, where a request to display a range ofevents is received. Continuing the above example, the global orderedlist includes 145 fully qualified event references. A request may bereceived to display the last 5 events, the second 50 events, the firstevent, all 145 of the events, or any other sub-range of the events.

The process proceeds to block 410, where event data is requested fromone or more of the computing devices based on the range of eventreferences requested from the global ordered list. For example, if thefirst 50 events are requested, then the first 50 entries in the globalordered list are retrieved. Continuing the example above, the first 7events from device A would be requested, all 37 of the events fromdevice B would be requested, and the first 6 events from device C wouldbe requested. Thus a total of 50 events are retrieved from threedifferent computing devices, without retrieving any unnecessary events.In one embodiment these requests are made in parallel, however requestsmay be submitted to individual devices serially. Also, in oneembodiment, a range of events may be requested from a single computingdevice in a single network transaction, however requests may also bemade individually.

The process proceeds to block 412, where the raw data is displayed. Inone embodiment, event data retrieved from individual computing devicesare displayed according to the global order. In one embodiment, therequested raw data is displayed with the one or more analysis results.In this way, a user may see the analysis results as well as portions ofthe underlying data. The process then proceeds to a return block.

FIG. 5 illustrates a logical flow diagram generally showing oneembodiment of a process an individual computing device may perform inthe course of performing a distributed search query. In one embodiment,process 500 is performed by one of client devices 101-103.

Process 500 begins, after a start block, at block 502, where a sub-queryis received from a distributed search server. The process then proceedsto block 504, where data such as events are analyzed according to thereceived sub-query. In one embodiment, as events are analyzed, eventsthat contribute to the requested analysis are referenced in a list ofevent references.

The process then proceeds to block 506, where the results of theanalysis and the list of event references are transmitted to thecomputing device that submitted the sub-query. In one embodiment, thisdevice is distributed search server 109.

The process then proceeds to block 508, where a request for one or morepieces of event data is received. In one embodiment, the requestincludes a contiguous range of event data. In another embodiment,individual pieces of event data are individually requested. The processthen proceeds to block 510, where the requested pieces of event data aretransmitted to the requestor. The process then proceeds to a returnblock.

FIG. 6 illustrates one non-limiting example of an interactive report600; however, other layouts containing other types of information aresimilarly contemplated. The interactive report was generated based on asearch query 602 of all domain name system (dns) lookups the specifiedclients performed “yesterday”. The report is broken into threesections—a timeline 604, a field picker 606, and an event data view 608.The timeline includes a bar graph 610 depicting how many dns lookupswere performed each hour. Field picker 606 is generally used to selectfields 612 from all of the fields available on a given type of event. Inthis example, field picker 606 has been used to select two of the 24fields associated with dns lookup events: client host and client IP.Thus, the event data displayed in the event data view will contain onlythese two fields. Finally, the event data view 608 displays raw eventdata, currently 50 results per page. A total of 562 events were gatheredfrom 79 clients. However, only the first 50 events have been downloadedto the distributed search server at the time this display was generated.If the user were to select another range of 50 events, the distributedsearch server could retrieve these 50 events from one or more of theclients in real-time as discussed above in conjunction with FIGS. 4 and5.

It will be understood that figures, and combinations of steps in theflowchart-like illustrations, can be implemented by computer programinstructions. These program instructions may be provided to a processorto produce a machine, such that the instructions, which execute on theprocessor, create means for implementing the actions specified in theflowchart block or blocks. The computer program instructions may beexecuted by a processor to cause a series of operational steps to beperformed by the processor to produce a computer implemented processsuch that the instructions, which execute on the processor to providesteps for implementing the actions specified in the flowchart block orblocks. These program instructions may be stored on a computer readablemedium or machine readable medium, such as a computer readable storagemedium.

Accordingly, the illustrations support combinations of means forperforming the specified actions, combinations of steps for performingthe specified actions and program instruction means for performing thespecified actions. It will also be understood that each block of theflowchart illustration, and combinations of blocks in the flowchartillustration, can be implemented by modules such as special purposehardware-based systems which perform the specified actions or steps, orcombinations of special purpose hardware and computer instructions.

The above specification, examples, and data provide a completedescription of the manufacture and use of the composition of thedescribed embodiments. Since many embodiments can be made withoutdeparting from the spirit and scope of this description, the embodimentsreside in the claims hereinafter appended.

1. A method, comprising: sending, to a plurality of computing systems, asearch request to search a set of time stamped events; receiving, fromtwo or more computing systems among the plurality of computing systems,at least two sets of search results and event references, wherein eachset of search results of the at least two sets of search resultsincludes information associated with one or more time stamped eventsthat satisfy the search request, wherein the event references include anevent identifier corresponding to each of the one or more time stampedevents that satisfy the search request; aggregating the at least twosets of search results into an aggregated search result; causinggraphical display of information based on the aggregated search result;in response to receiving input requesting additional information from atleast a portion of the aggregated search result, determining a set ofevent identifiers corresponding to time stamped events included in theat least a portion of the aggregated search result; sending, to the twoor more computing systems, a request for time stamped eventscorresponding to the set of event identifiers; receiving, from the twoor more computing systems, the requested time stamped events; andcausing display of the requested additional information using thereceived requested time stamped events.
 2. The method of claim 1,wherein the event identifier indicates a computing system that stores atime stamped event and a value indicating a location of the time stampedevent in the computing system.
 3. The method of claim 1, wherein the twoor more computing systems execute the search request to search the timestamped events concurrently, wherein each of the two or more computingsystems is responsible for searching a portion of the time stampedevents.
 4. The method of claim 1, wherein the search result informationindicates statistical information about one or more time stamped eventsthat satisfy the search request.
 5. The method of claim 1, furthercomprising: wherein the search request is periodically sent to thecomputing system; generating a report based on search results from theperiodic search request.
 6. The method of claim 1, wherein the searchrequest is sent to the computing system in response to an eventoccurrence.
 7. The method of claim 1, wherein the set of time stampedevents include raw data.
 8. The method of claim 1, wherein the receivedrequested time stamped events include raw data.
 9. An apparatus,comprising: a subsystem, implemented at least partially in hardware,that sends, to a plurality of computing systems, a search request tosearch a set of time stamped events; a subsystem, implemented at leastpartially in hardware, that receives, from two or more computing systemsamong the plurality of computing systems, at least two sets of searchresults and event references, wherein each set of search results of theat least two sets of search results includes information associated withone or more time stamped events that satisfy the search request, whereinthe event references include an event identifier corresponding to eachof the one or more time stamped events that satisfy the search request;a subsystem, implemented at least partially in hardware, that aggregatesthe at least two sets of search results into an aggregated searchresult; a subsystem, implemented at least partially in hardware, thatcauses graphical display of information based on the aggregated searchresult; a subsystem, implemented at least partially in hardware, that inresponse to receiving input requesting additional information from atleast a portion of the aggregated search result, determines a set ofevent identifiers corresponding to time stamped events included in theat least a portion of the aggregated search result; a subsystem,implemented at least partially in hardware, that sends, to the two ormore computing systems, a request for time stamped events correspondingto the set of event identifiers; a subsystem, implemented at leastpartially in hardware, that receives, from the two or more computingsystems, the requested time stamped events; and a subsystem, implementedat least partially in hardware, that causes display of the requestedadditional information using the received requested time stamped events.10. The apparatus as recited in claim 9, wherein the event identifierindicates a computing system that stores a time stamped event and avalue indicating a location of the time stamped event in the computingsystem.
 11. The apparatus as recited in claim 9, wherein the two or morecomputing systems execute the search request to search the time stampedevents concurrently, wherein each of the two or more computing systemsis responsible for searching a portion of the time stamped events. 12.The apparatus as recited in claim 9, wherein the search resultinformation indicates statistical information about one or more timestamped events that satisfy the search request.
 13. The apparatus asrecited in claim 9, further comprising: wherein the search request isperiodically sent to the computing system; a subsystem, implemented atleast partially in hardware, that generates a report based on searchresults from the periodic search request.
 14. The apparatus as recitedin claim 9, wherein the search request is sent to the computing systemin response to an event occurrence.
 15. The apparatus as recited inclaim 9, wherein the set of time stamped events include raw data. 16.The apparatus as recited in claim 9, wherein the received requested timestamped events include raw data.
 17. A non-transitory computer-readablemedium storing one or more sequences of instructions, wherein executionof the one or more sequences of instructions by one or more processorscauses the one or more processors to perform: sending, to a plurality ofcomputing systems, a search request to search a set of time stampedevents; receiving, from two or more computing systems among theplurality of computing systems, at least two sets of search results andevent references, wherein each set of search results of the at least twosets of search results includes information associated with one or moretime stamped events that satisfy the search request, wherein the eventreferences include an event identifier corresponding to each of the oneor more time stamped events that satisfy the search request; aggregatingthe at least two sets of search results into an aggregated searchresult; causing graphical display of information based on the aggregatedsearch result; in response to receiving input requesting additionalinformation from at least a portion of the aggregated search result,determining a set of event identifiers corresponding to time stampedevents included in the at least a portion of the aggregated searchresult; sending, to the two or more computing systems, a request fortime stamped events corresponding to the set of event identifiers;receiving, from the two or more computing systems, the requested timestamped events; and causing display of the requested additionalinformation using the received requested time stamped events.
 18. Thenon-transitory computer-readable medium as recited in claim 17, whereinthe event identifier indicates a computing system that stores a timestamped event and a value indicating a location of the time stampedevent in the computing system.
 19. The non-transitory computer-readablemedium as recited in claim 17, wherein the two or more computing systemsexecute the search request to search the time stamped eventsconcurrently, wherein each of the two or more computing systems isresponsible for searching a portion of the time stamped events.
 20. Thenon-transitory computer-readable medium as recited in claim 17, whereinthe search result information indicates statistical information aboutone or more time stamped events that satisfy the search request.
 21. Thenon-transitory computer-readable medium as recited in claim 17, furthercomprising: wherein the search request is periodically sent to thecomputing system; generating a report based on search results from theperiodic search request.
 22. The non-transitory computer-readable mediumas recited in claim 17, wherein the search request is sent to thecomputing system in response to an event occurrence.
 23. Thenon-transitory computer-readable medium as recited in claim 17, whereinthe set of time stamped events include raw data.
 24. The non-transitorycomputer-readable medium as recited in claim 17, wherein the receivedrequested time stamped events include raw data.